Add ALL_DONE_MIN_ONE_SUCCESS trigger rule#53959
Conversation
Lee-W
left a comment
There was a problem hiding this comment.
I'm not completely against it, but I think we could use something like a branch operator with all_done. -0 to this, but would love to hear how everyone think 🙂
There is no branching here. I don't know of any simple way to get the desired behavior. you need to workaround with additional tasks which in big environments creates many unnecessary tasks (more load on the scheduler etc..) |
shahar1
left a comment
There was a problem hiding this comment.
Overall LGTM - I could imagine use cases of this trigger rule.
For the long term, maybe we should think about changing the trigger rule mechanism so the definition could be more flexible instead of the hardcoded enum (e.g., this TR will be represented as something like trigger_rule={'done': 'all', 'success': '>=1'}).
We can look into it after Airflow 3.1 if you have a proposal feel free to suggest it in the mailing list. |
|
Yep, what's in my mind is customizable trigger rules as well. @shahar1 let's see whether we can work on something together after 3.1 🙂 |
We should be careful with that. Trigger rules are on the hottest path of scheduler, and we should not have too many of them, also customizability of it is limited as it runs in scheduler. We should generally try to minimise and rather hard-code all the logic here I think, |
|
Hey @eladkal, |
This trigger rule is useful for cases where we must verify all upstream is done with execution but we only need one of them to be successful.
Verified also with example dag: